feat(invoker,network): add deployment metrics and connection pool instrumentation#4567
feat(invoker,network): add deployment metrics and connection pool instrumentation#4567robertream wants to merge 7 commits into
Conversation
* Deployment-level observability to the invoker subsystem, combining issues restatedev#4553 and restatedev#4454 into a single change. Lets operators distinguish slow user services from internal Restate bottlenecks. New metrics: - restate.invoker.http_request_duration.seconds (TTFB) - restate.invoker.http_total_duration.seconds (total duration) - restate.invoker.queue_duration.seconds (internal wait time) - restate.invoker.active_invocations (concurrency gauge) - restate.invoker.http_status_code.total (non-200 responses) - restate.invoker.throttle_balance (token bucket debt) * Label enrichment on invocation_tasks, task_duration, eager_state_truncated, enqueue, and concurrency metrics with partition_id, service_name, and deployment_id where applicable. * ServiceMetrics struct with gauge/counter/histogram helpers.All service metric recording goes through the struct methods including TaskMetrics struct handles INVOCATION_TASKS lifecycle counters with status/transient labels. * ResponseStream::instrument(metric) builder for HTTP timing, enqueued_at timestamp on InvokeCommand for queue duration, and event-driven throttle balance recording at slot acquire/release. * LazyIntern for generic string interpolation including STATUS_CODE_LOOKUP for zero-allocation HTTP status code interning.
…v#4559) Add comprehensive metrics to the networking layer's connection pool via two scoped structs: NetworkMetrics (swimlane-scoped, pre-cached per variant): - pool.connections.active gauge - connection.pending gauge - connection.acquisition.duration histogram (cache=hit/miss, result) - connection.handshake.duration histogram (result=success/error) - rpc.duration histogram (result=success/error) ConnectionMetrics (swimlane + peer_name + direction, stored on Connection): - connection.opened/closed counters (+ legacy connection_created/dropped) - permit_acquisition.duration histogram - connection.duration (lifetime) histogram Key design decisions: - NetworkMetrics instances pre-built in a static array indexed by Swimlane - ConnectionMetrics created via NetworkMetrics::connection(peer, direction) - All labels are &'static str via LazyIntern for peer IDs — zero allocation - Legacy metrics (connection_created/dropped) preserved with enriched labels - Pool gauge decremented in deregister() to avoid double-count on drain+drop - LazyIntern extracted to restate_core::metric_definitions for reuse Also renames invoker metric constants to follow OTel naming conventions.
… import Clippy flags new_without_default on LazyIntern<K>::new(). Add the trivial Default impl. Also remove an unused std::sync::LazyLock import in the network metric_definitions introduced by the connection pool instrumentation commit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instrument ServiceDiscovery with counters and histograms for deployment
endpoint reachability — currently only visible in logs. Three metrics:
- restate.deployment.discovery.attempts.total{outcome}
(success | retryable_exhausted | permanent_failure)
- restate.deployment.discovery.retries.total{reason}
(server_error | rate_limited | connection | body_error | other)
- restate.deployment.discovery.duration.seconds{outcome}
describe_metrics() called from ServiceDiscovery::new(), following the
crate-internal pattern used across the codebase.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Nice contribution — closing a real observability gap. A few notes: LazyIntern::get double-lookup: The initial self.cache.get(key) before self.cache.entry(key.clone()) is redundant — DashMap::entry() already handles the hit case internally, so the pre-check just adds an extra hash per miss. The entry() call alone is sufficient. INVOKER_CONCURRENCY_LIMIT rename: Renaming to INVOKER_CONCURRENCY_SLOTS_LIMIT will silently break existing dashboards and alerts. Consider keeping the original metric name and only adding the new labels on top. deployment_id label cardinality split: Emitting metrics without deployment_id before the deployment is pinned, then with it after, creates two distinct Prometheus time series for the same task. Using a placeholder like "unknown" would keep the label set uniform throughout. |
- Remove redundant DashMap pre-check in LazyIntern::get; entry() handles hits - Restore original metric name restate.invoker.concurrency_limit to avoid breaking dashboards - Use "unknown" placeholder for deployment_id label instead of omitting it, keeping Prometheus label sets uniform Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Thanks for the review @makarandg2510-a11y! All three points addressed in 406e11b:
|
…helper
Remove the #[cfg(test)] EMPTY const from ServiceMetrics and introduce
local test_metrics() helpers that use realistic fake labels ("test",
"test.Service") via ServiceMetrics::new().
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ilures Use empty string as deployment_id sentinel for "not yet pinned" so the decrement guard works correctly, while emitting "unknown" in Prometheus labels to keep label sets uniform. Also add comment explaining legacy invoker_id label in quota metrics. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Summary
Deployment-level observability to the invoker subsystem, combining
issues #4553 and #4454 into a single change. Lets operators distinguish
slow user services from internal Restate bottlenecks.
Invoker metrics
New metrics:
Label enrichment on invocation_tasks, task_duration, eager_state_truncated,
enqueue, and concurrency metrics with partition_id, service_name, and
deployment_id where applicable.
ServiceMetrics struct with gauge/counter/histogram helpers. All service
metric recording goes through the struct methods including TaskMetrics
struct handles INVOCATION_TASKS lifecycle counters with status/transient
labels.
ResponseStream::instrument(metric) builder for HTTP timing, enqueued_at
timestamp on InvokeCommand for queue duration, and event-driven throttle
balance recording at slot acquire/release.
Connection pool instrumentation (#4559)
Two scoped metric structs for the networking layer:
NetworkMetrics (swimlane-scoped, pre-cached per variant):
pool.connections.activegaugeconnection.pendinggaugeconnection.acquisition.durationhistogram (cache=hit/miss, result=success/error)connection.handshake.durationhistogram (result=success/error)rpc.durationhistogram (result=success/error)ConnectionMetrics (swimlane + peer_name + direction, stored on Connection):
connection.opened/closedcounters (+ legacy connection_created/dropped)permit_acquisition.durationhistogramconnection.duration(lifetime) histogramKey design decisions:
NetworkMetrics::connection(peer, direction)&'static strviaLazyIntern— zero allocation on emitLazyInternextracted torestate_core::metric_definitionsfor reuseVerification
cargo checkpassescargo clippy --all-targets -- -D warningscleancargo fmt --all -- --checkcleancargo nextest run -p restate-invoker-impl -p restate-core— 65/65 passingCloses #4553
Closes #4454
Closes #4559